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Abstract 

Building  a  distributed  publish/ subscribe  infrastructure  amounts  to  defining  a  service  model  (or  in¬ 
terface)  and  providing  an  implementation  for  it.  A  typical  distributed  implementation  is  architected  as 
a  network  of  dispatcher  components,  each  one  implementing  appropriate  protocols  and  algorithms,  that 
collectively  realize  the  chosen  service  model.  The  service  model  should  provide  a  value-added  service  for 
a  wide  variety  of  applications,  while  the  implementation  should  gracefully  scale  up  to  handle  an  intense 
traffic  of  publicahons  and  subscriptions.  We  believe  that  the  design  of  such  service  models  and  implemen¬ 
tations  must  be  guided  by  a  systematic  evaluation  method,  which  in  turns  must  be  based  on  a  carefully 
chosen  benchmark  suite.  In  this  paper,  we  lay  out  a  set  of  requirements  for  a  benchmark  suite  for  distributed 
publish/ subscribe  services,  and  we  outline  its  primary  components.  The  ideas  proposed  in  this  paper  are 
based  on  our  own  experience  in  building  and  studying  publish/ subscribe  infrastructures,  and  on  existing 
evaluation  frameworks. 
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1  Evaluating  a  Distributed  Publish/Subscribe  Service 

Within  a  publish/ subscribe  service,  information  flows  from  publishers  fo  subscribers  according  fo  fhe  spe¬ 
cific  selecfion  criferia  expressed  by  individual  subscribers.  Subscribers  express  fheir  inferesfs  by  means  of 
subscripfions,  while  publishers  simply  publish  informafion.  The  service  accepfs  subscriptions  and  publica¬ 
tions,  and  relays  publications  fo  subscribers  fhaf  declared  mafching  subscripfions.  The  general  archifecfure 
of  a  disfribufed  publish/ subscribe  infrasfrucfure  is  a  nefwork  of  dispafchers  [1,  4,  5,  7].  Publishers  and 
subscribers  are  direcfly  connecfed  fo  one  dispafcher,  fo  which  fhey  send  fheir  subscripfions  and  publi¬ 
cations.  Every  dispafcher  processes  subscripfions  and  publications  according  fo  some  protocol,  possibly 
redisfribufing  subscripfions  fo  ofher  adjacenf  dispafchers  and  publications  fo  adjacenf  dispafchers  and/or 
subscribers. 

Realizing  a  publish/ subscribe  service  according  fo  fhis  general  schema  amounfs  fo  designing 

•  a  data  model  for  publishable  informafion  (i.e.,  formaf,  sfrucfure,  f5rpes,  efc.) 

•  fhe  selection  mechanisms  available  fo  subscribers  (i.e.,  scope,  S5mfax,  and  semantics  of  subscripfions) 

•  fhe  topology  of  fhe  nefwork  of  dispafchers,  including  fhe  rules  by  which  dispafchers  join  and  leave 
fhe  nefwork,  fhe  profocols  used  by  dispafchers  fo  esfablish  a  connection,  and  possibly  fo  aufhenficafe 
each  ofher,  efc. 

•  fhe  communication  protocols  used  among  dispafchers,  and  befween  dispafchers  and  clienfs.  These  pro¬ 
focols  cover  fhe  basic  exchange  of  publicafions  and  subscripfions  as  well  as  all  fhe  auxiliary  informa¬ 
fion  flows  needed  fo,  say,  compufe  shorfesf  pafhs  and  ofher  such  specific  funcfions. 

•  fhe  internal  architecture  of  dispatchers,  including  physical  componenfs,  such  as  inpuf/oufpuf  modules, 
swifching  componenfs,  and  logic  componenfs,  including  mafching  algorifhms,  roufing  algorifhms, 
and  fheir  dafa  sfrucfures. 

Each  and  every  one  of  fhese  design  aspecfs  has  been  extensively  sfudied,  in  various  forms  and  combi- 
nafions,  in  differenf  confexfs,  ranging  from  dafabases  fo  programming  languages,  fo  networking  research. 
The  ulfimafe  challenge  for  fhe  designer  of  a  disfribufed  publish/ subscribe  service  is  fo  engineer  exisfing 
and  new  fechniques  fo  create  a  publish/ subscribe  infrasfrucfure  capable  of  offering  an  added-value  ser¬ 
vice  for  fhe  widesf  variefy  of  applicafions,  and  af  fhe  same  time,  capable  of  scaling  from  simple  localized 
applications,  up  fo  complex,  fraffic-infensive,  highly  disfribufed  applicafions. 

Clearly,  fhese  goals  demand  a  mefhodical  engineering  approach.  Specifically,  we  believe  fhaf  fhe  nu¬ 
merous  degrees  of  freedom  in  fhe  design  space  and  fhe  inberenf  complexify  of  each  aspecf  of  fhe  design 
emphasize  fhe  imporfance  of  serious  evaluation  mefhods.  Even  more  fo  fhe  poinf,  we  argue  fhaf  nof  only 
mefhodical  performance  evaluation  and  service  validafion  are  necessary  sfeps  in  fhe  design  process,  buf 
also  fhaf  fhey  should  be  considered  as  fhe  primary  guides  for  fhe  developmenf  and  infegrafion  of  technolo¬ 
gies  for  disfribufed  publish/ subscribe  services. 

Whaf  we  propose  as  an  initial  solution  fo  fhis  design  problem  is  fhe  definifion  of  a  benchmark  suite. 
In  fhe  resf  of  fhe  paper  we  lay  ouf  a  sef  of  requiremenfs  and  some  inifial  specificafions  for  a  disfribufed 
publish/ subscribe  benchmark  suite. 


2  Requirements  for  a  Benchmark  Suite 

The  firsf  requiremenf  for  a  benchmark  suite  is  in  facf  a  mefa-requiremenf,  or  rafher  a  condition  on  fhe 
process  by  which  fhe  benchmark  is  defined.  If  is  crucial  fhaf  fhe  benchmark  suife  be  widely  accepfed  by 
researchers  and  practitioners  in  fhe  field,  and  consequenfly  adopted  as  parf  of  fheir  design  environmenf. 
If  is  fherefore  very  imporfanf  fhaf  fhe  formulation  of  fhe  benchmark  ifself  be  a  communal  acfivify.  In  fhis 
spirif,  we  infend  fhis  paper  as  an  inifial  proposal,  mean!  fo  generate  discussion  and  inferesf,  which  we  hope 
would  fhen  franslafe  info  a  cooperative  benchmark  definifion  efforf. 

The  specific  high-level  requiremenfs  for  fhe  benchmark  suife  are  direcfly  relafed  fo  fhe  funcfion  of  fhe 
benchmark.  As  we  said,  fhe  funcfion  of  fhe  benchmark  can  be  broken  down  info  two  main  goals: 
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•  validation  of  the  service  interface,  and 

•  performance  evaluation. 

In  order  to  satisfy  the  first  goal,  the  benchmark  suite  must  include  tests  and  methods  that  assess  the 
suitability  of  a  particular  publish/ subscribe  service.  Suitability  is  intended  as  an  aggregate  measure  of 
the  added  value  provided  by  the  service  to  a  wide  range  of  applications.  Obviously,  this  evaluation  is 
characterized  by  qualitative  parameters,  and  is  likely  to  be  affected  by  subjective  interpretations.  Therefore, 
the  benchmark  should  guide  and  disambiguate  the  evaluation  process  by  establishing  a  detailed  capability 
model. 

The  second  goal  is  somewhat  more  concrete,  since  it  is  characterized  by  more  objective  and  quantitative 
measurements.  The  benchmark  should  be  based  on  the  cross-product  of  a  number  of  applications,  executed 
over  series  of  configurations  and  distributions  of  their  components,  under  a  series  of  workloads.  Perfor¬ 
mance  metrics  should  be  based  on  time  and  space  complexity.  Typical  such  metrics  are  end-to-end  latency, 
end-to-end  maximal  throughput,  and  maximal  footprint  for  individual  components. 

Despite  its  conceptual  simplicity,  a  performance  benchmark  poses  a  fundamental  obstacle,  when  com¬ 
bined  with  a  service  suitability  benchmark.  The  problem  is  that  an  ideal  service  suitability  benchmark 
should  not  be  biased  towards  any  specific  publish/subscribe  interface.  By  contrast,  a  performance  bench¬ 
mark  must  consist  of  concrete  application  programs,  which  must  use  specific  service  interfaces.  One 
extreme  approach  to  resolve  this  conflict  would  be  to  define  a  specific  interface,  as  an  integral  part  of 
the  benchmark,  and  to  focus  on  applications  that  use  that  interface.  In  this  case,  a  designer  of  a  pub¬ 
lish/subscribe  system  with  a  different  service  interface  would  have  to  provide  an  adapter  that  would  some¬ 
how  translate  the  benchmark  service  interface  to  his  or  her  own  service  interface.  The  opposite  approach 
would  be  to  build  the  benchmark  out  of  very  service-neutral  applications,  or  better,  applications  capable  of 
adapting  to  different  service  models. 

Neither  of  these  approaches  seems  satisfactory.  The  second  one  is  probably  not  feasible  at  all,  since 
arguably  there  is  no  such  thing  as  a  real  service-neutral  application.  While  the  first  one  poses  another 
dilemma  in  choosing  the  t5q)e  of  benchmark  interface:  a  "common  denominator"  interface  would  probably 
be  too  simplistic,  whereas  a  very  rich  interface  would  penalize  most  services.  In  any  case,  using  a  single 
interface  as  a  reference  would  create  a  restricted  and  biased  benchmark.  The  natural  compromise  that  we 
propose  is  to  select  applications  that  use  a  number  of  specific  publish/subscribe  interfaces.  The  benefit  of 
this  idea  is  that  it  would  create  a  fair  and  more  generic  benchmark.  The  drawback  is  that,  in  order  to  execute 
benchmark  applications,  the  service  designer  would  have  to  provide  several  adaptation  layers. 

The  choice  of  applications,  configurations,  and  workloads  is  clearly  crucial  for  the  formulation  of  the 
benchmark.  We  see  two  distinct  requirements  for  this  part  of  the  benchmark  suite.  On  one  hand,  part  of  the 
benchmark  must  be  an  expression  of  real,  current  applications,  independently  developed  by  third-party  or¬ 
ganizations.  Similarly,  configurations  and  workloads  must  reflect  real  computing  environments  and  actual 
use  scenarios.  Ideally,  this  part  of  the  benchmark  would  be  derived  from  recordings  of  actual  application 
sessions.  The  objective  of  this  section  of  the  benchmark  is  to  directly  benefit  current  technologies  and  usage 
patterns.  On  the  other  hand,  the  benchmark  should  also  account  for  future  application  developments  and 
unforeseen  usage  scenarios.  This  part  of  the  benchmark  suite  must  rely  on  S5mthetic  scenarios.  These  sce¬ 
narios  need  not  be  related  to  actual  usage  patterns,  and  should  instead  explore  situations  that  are  far  from 
the  current  realm  of  publish/ subscribe  applications. 


3  Initial  Benchmark  Specification 

Following  the  requirements  we  set  in  Section  2,  we  propose  a  benchmark  suite  composed  of  three  sections: 
an  interface  suitability  section,  an  applications  section,  and  a  synthetic  scenarios  section.  We  will  discuss  the 
details  of  every  one  of  these  sections  in  the  following. 

3.1  Interface  Suitability 

The  purpose  of  this  section  of  the  benchmark  is  to  establish  a  capability  model  for  publish/ subscribe  ser¬ 
vices.  The  benchmark  program  is  a  simple  semi-automated  self-evaluation  system  based  on  questions  and 
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answers.  The  program  guides  the  designer  in  the  evaluation  process  by  asking  questions  derived  from  the 
capability  model.  It  then  annotates  the  designer's  answers  and  eventually  computes  aggregate  evaluation 
metrics  based  on  coverage  criteria  expressed  in  the  capability  model. 

A  complete  formulation  of  fhe  capabilify  model  is  well  beyond  fhe  scope  of  fhis  paper,  and  as  we  said, 
if  should  resulf  from  a  joinf  efforf  of  researchers  and  pracfifioners  in  fhe  field.  Here  we  lisf  a  number  of 
general  feafures  fhaf  we  believe  should  be  included  in  fhe  capabilify  model.  Elemenfs  of  fhis  lisf  have  been 
exfensively  sfudied  wifhin  ofher  evaluation  frameworks  [4,  5, 2,  8]. 

•  publication  model:  dafa  model  for  publisheable  dafa.  This  model  should  classify  services  according  fo 
fhe  following  paramefers: 

-  structure:  characferizes  fhe  sfrucfure  of  nofificafions.  Typical  publications  can  be  classified  as: 
imsfrucfured,  lisfs  of  sfrings,  record-like  sfrucfures  wifh  posifional  or  name-based  identification 
of  affribufes,  recursive  sfrucfures,  such  as  LISP  expressions  or  XML  documenfs,  and  composife 
publications,  made  of  digesfs  of  ofher  publications 

-  types:  predefined  domains  of  values.  Typical  t5rpe  classifications  would  be  binary  or  sfring,  sim¬ 
ple  atomic  f5q)es  (such  as  infegers,  dales,  booleans),  and  fyped  sfrucfures,  fhaf  is,  sfrucfures 
whose  combinafion  of  fields  consfifufe  a  t5q)e  in  ifself 

-  limits:  fofal  byfe  size,  number  of  affribufes,  limifs  for  f5rpes  (sfring  lengfh,  integer  sizes  or  ranges 
of  values),  and  number  and  depfh  of  sub-sfrucfures 

•  subscription  model:  defines  fhe  selecfion  capabilities  of  fhe  publish/ subscribe  service: 

-  scope:  defines  whaf  parfs  of  a  publicafion  can  be  evaluated  and  selected  wifhin  subscriptions.  A 
fypical  classification  of  scope  may  be  fhe  following:  single  globally  known  field,  single  subscrip¬ 
tion-specific  field,  limbed  number  of  fileds,  entire  publications,  and  groups  of  correlafed  publi¬ 
cations 

-  language  power:  characferizes  fhe  language  fhaf  defines  subscription  in  ferms  of  ifs  expressive 
power.  Typical  language  power  classifications  are:  simple  predicafes,  such  as  equably,  inequalify, 
and  sfring  operafors,  and  composife  expressions 

-  language  style:  declarafive,  imperafive 

-  other  features:  exfensibilify  (for  example,  by  means  of  plug-ins),  useful  special  operafors  such  as  a 
"cerfificafe-based  aufhenficafion"  predicafe  fhaf  would  selecf  all  fhe  publicafion  fhaf  a  clienf  can 
successfully  aufhenficafe. 

•  interface  access  methods:  supporf  for  local  access,  for  example,  wifh  shared  memory  or  ofher  OS  in¬ 
memory  connecfions  such  as  UNIX  sockefs  or  signals,  and  remofe  access,  wifh  specific  communica¬ 
tion  profocols 

•  portability:  supporf  for  multiple  plafforms  and  language  bindings 

•  service  model:  examples  are  reliable/unreliable  for  poinf-fo-poinf  communications,  end-fo-end  reli¬ 
able/  unreliable,  qualify  of  service  negofiafion,  persisfenf  sfore-and-forward  service 

•  auxiliary  features:  for  example,  securify  and  supporf  for  mobilify 

3.2  Applications 

This  section  of  fhe  benchmark  suife  is  inf  ended  fo  evaluafe  fhe  performances  of  fhe  publish/ subscribe 
mfrasfrucfure,  when  serving  currenf  common  applicafions,  under  observed  fraffic  loads.  For  fhese  bench¬ 
marks,  we  propose  fo  consider  applicafions  wifhin  fhe  following  categories: 

•  internet-based  trading:  fhis  includes  auction  systems  (such  as  eBay),  airline  reservation  systems,  and 
ofher  generic  e-commerce  applicafions.  This  class  of  applicafions  is  characferized  by  high  volumes  of 
fraffic,  wifh  sparse  subscribers  over  a  wide-area  network.  Subscribers  are  likely  to  select  very  specific 
information  ouf  of  a  varied  information  space 
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•  news:  network-based  sportscast  systems,  financial  news  systems,  emergency  armouncements,  and 
general  event  notification.  Clearly,  this  is  also  a  class  of  applicafions  fhaf  span  wide-area  networks. 
However,  in  contrast  with  internet  applications,  it  is  likely  to  have  more  dense  distributions  of  sub¬ 
scribers,  wifh  more  generic  and  /  or  less  volafile  requesfs,  wifh  a  higher  fraffic  flow 

•  networked  interactive  games:  fhis  class  of  applicafions  is  inferesfing  for  ifs  essential  real-time  require- 
menfs,  as  well  as  for  fhe  very  infense,  buf  probably  localized,  fraffic 

•  software  systems:  fhis  cafegory  includes  workflow  managemenf  sysfems,  application  managemenf, 
disfribufed  agenda  managemenf,  and  mobile  agenfs  plaf forms.  Application  from  fhis  cafegory  are 
likely  fo  require  fhe  exchange  of  complex  objecfs,  and  fherefore  are  likely  fo  exploif  fhe  mosf  advanced 
dafa  modeling  and  selection  feafures  of  fhe  publish/ subscribe  service. 

These  benchmarks  also  define  fhe  precise  workload  and  fhe  precise  componenf  configurations,  in  com- 
binafion  wifh  each  individual  application.  We  propose  fo  define  fhese  workloads  and  configurations  by 
recording  several  acfual  sessions  for  each  applicafion.  We  realize  fhaf  fhis  may  be  a  considerable  efforf  in 
ifself,  and  fhaf  fhe  measuremenf  process  is  likely  fo  pose  serious  soffware  engineering  and  measuremenf 
challanges. 

3.3  Synthetic  Scenarios 

S5mfhefic  benchmarks  are  meanf  fo  fesf  a  publish/ subscribe  service  imder  uncommon  workloads,  or  under 
fraffic  patterns  and  configurations  nof  exhibifed  by  currenf  applicafions.  S5mfhefic  benchmarks  should 
also  focus  on  specific  feafures  or  fraffic  pafferns,  since  fhis  can  nof  be  easily  done  wifh  workloads  derived 
from  acfual  applicafions.  The  use  of  S5mfhefic  workload  generators  is  common  pracfice  in  fhe  evaluafion  of 
nefwork  profocols  and  network  architectures.  S5mthetic  scenarios  have  also  already  been  used  to  evaluate 
a  number  of  techniques  for  publish/ subscribe  services  [4,  6,  3]. 

Being  acfual  programs,  S5mfhefic  benchmarks  too  are  designed  fo  use  a  specific  service  inferface.  There¬ 
fore,  nof  every  benchmark  program  may  be  applicable  fo  each  and  every  service  inferface.  This  section 
of  fhe  benchmark  suife  musf  confain  enough  benchmark  programs  fo  cover  fhe  mosf  common  service  in- 
ferfaces.  In  fhe  following,  we  will  presenf  an  example  of  a  S5mfhefic  benhcmark  fhaf  uses  a  specific  pub¬ 
lish/subscribe  service  model,  nonfheless  fhe  basic  ideas  expressed  by  fhe  example  are  valid  for  ofher  service 
models  as  well. 

The  benchmark  is  essenfially  composed  of  fwo  parfs: 

•  topology:  fhis  parf  defines  fhe  disfribufion  of  service  componenfs  and  applicafion  componenfs  over  a 
nefwork.  The  fopology  is  expressed  in  ferms  of  fhe  connecfions  between  components,  and  their  costs. 

•  application  behavior:  this  part  defines  fhe  combinafion  of  service  requesfs,  including  specific  values  for 
publications  and  subscripfions. 

The  fopology  may  be  defined  direcfly  af  fhe  applicafion  level  or  as  an  overlay  of  nefwork  and  applica¬ 
fion  fopologies.  The  subsfrafe  nefwork  fopology  can  be  generated  using  an  appropriate  random  graph 
model  [9].  If  is  unclear  fo  us  whefher  fhe  same  f5q3e  of  models  can  be  adapfed  fo  generate  plausible 
application-level  fopologies  (we  were  unable  fo  find  a  relevanf  sfudy  of  application-level  fopologies  in 
fhe  liferafure).  If  fhe  subsfrafe  is  a  network-level  topology,  an  application  interconnection  must  be  set  up 
on  top  of  fhe  subsfrafe  nefwork.  This  can  be  done  in  a  number  of  ways.  Whaf  we  propose  is  an  incremenfal 
process,  which  works  as  follows:  fhe  number  of  application-level  nodes  is  given  as  an  inifial  parameter. 
The  firsf  "roof"  application-level  componenf  is  allocated  on  a  randomly  chosen  node  of  fhe  subsfrafe.  The 
remaining  application-level  nodes  are  randomly  placed  and  randomly  connecfed  fo  a  number  of  exisfing 
componenfs.  The  way  fhese  fwo  random  choices  are  made  depends  in  parf  on  fhe  t5rpe  of  service  archifec- 
fure,  and  in  parf  on  ofher  benchmark  fopology  parameters.  For  example,  if  fhe  service  requires  an  acyclic 
archifecfure,  fhen  new  nodes  will  be  connecfed  fo  exacfly  one  exisfing  node.  The  remaining  random  pa- 
ramefers  can  be  adjusfed  by  using  disfribufions  fhaf  favor  localify  in  bofh  placemenf  and  interconnections. 
Once  fhe  service  fopology  is  defined,  fhe  benchmark  musf  allocate  applicafion  componenfs  on  each  hosf. 
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This  process  can  follow  the  same  schema  described  above,  again  maintaining  some  reasonable  form  of 
locality  in  connecting  applications  to  service  endpoints. 

The  next  step  in  the  formulation  of  this  sample  S5mthetic  benchmark  consists  in  assigning  behaviors  to 
application  components.  This  part  of  the  benchmark  is  characterized  by  several  degrees  of  freedom,  and 
therefore  lots  of  parameters.  What  we  give  here  is  a  list  of  parameters  that  subsumes  a  generic  random¬ 
ized  behavior.  Specifically,  we  assume  periodic  event  generators,  as  well  as  periodic  subscribers.  That  is, 
publishers  that  emit  publications  at  a  fixed  rate,  and  subscribers  that  continuously  subscribe  (for  some  in¬ 
formation),  receive  a  number  of  publications,  unsubscribe,  wait  some  time,  and  then  re-subscribe.  These 
are  the  essential  parameters: 

•  publication  rate:  number  of  publications  issued  per  time  unit  (poisson  distribution) 

•  active  subscription  cycle  duration:  interval  during  which  a  subscriber  has  at  least  one  active  subscription. 
It  may  be  defined  by  the  number  of  received  publications,  or  by  a  simple  timeout 

•  inactive  subscription  cycle  duration:  interval  in  which  a  subscriber  has  no  active  subscriptions.  This  is  a 
simple  interval  between  unsubscriptions  and  subscriptions  in  a  subscriber's  cycle 

•  publication  model:  parameters  used  to  generate  publications  (remember  that,  in  this  example,  publica¬ 
tions  are  record-like  structures): 

-  number  of  fields:  this  should  also  be  randomly  determined  according  to  a  Poisson  distribution 
(since  the  value  can  never  be  less  than  one) 

-  namespace  for  attributes:  set  of  names  used  to  identify  attributes  in  notifications.  We  propose  to 
use  a  random  selection  out  of  a  dictionary  of  words  (e.g.,  a  standard  English  dictionary) 

-  prevalence  of  attribute  names:  probability  function  for  attribute  names  within  the  namespace 

-  prevalence  of  types:  probability  functions  for  attribute  t5^es 

-  values  space:  set  of  values,  one  set  per  t5rpe.  This  parameter  defines  ranges  for  numeric  values, 
and  dictionaries  for  string  values 

-  prevalence  of  values:  probability  function  for  values 

•  subscription  model:  parameters  used  to  generate  subscriptions,  and  to  associate  them  with  subscribers. 
In  this  example,  subscriptions  are  conjunctions  of  elementary  predicates,  each  one  defined  over  the 
value  of  a  given  attribute.  Predicates,  like  attributes,  are  t5rped,  meaning  that  they  match  attributes  of 
a  specific  t5^e: 

-  number  of  subscriptions  per  subscriber:  a  subscriber  may  have  several  active  subscriptions  at  the 
same  time.  This  parameter  determines  the  distribution  of  subscriptions  over  subscribers 

-  number  of  predicates  per  subscription:  randomly  distributed 

-  name  space  for  subscriptions:  set  of  attribute  names  used  in  subscription  predicates.  This  set  is 
composed  of  words  extracted  randomly  from  a  dictionary  and  from  the  namespace  for  publica¬ 
tions 

-  percentage  of  shared  namespace:  indicates  what  fraction  of  the  namespace  of  subscriptions  is  ex¬ 
tracted  from  the  publications  namespace.  This  parameter  roughly  measures  the  degree  of  inte¬ 
gration  between  publishers  and  subscribers 

-  prevalence  of  attribute  names:  probability  function  for  names  in  subscription  predicates 

-  prevalence  of  types:  probability  function  for  t5rpes  in  predicates 

-  prevalence  of  predicate  operators  probability  function  for  operators  (equality,  greater-than,  lower- 
than,  etc.) 
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4  Conclusions 


We  are  convinced  that  systematic  performance  evaluation  and  service  validation  are  necessary,  crucial  steps 
in  the  design  of  publish/ subscribe  infrasfrucfures,  especially  fhose  based  on  disfribufed  archifecfures.  Wifh 
fhis  paper,  we  propose  fhe  formulafion  of  a  benchmark  suife  as  a  basis  for  fhaf  evaluafion  efforf.  We  also 
believe  fhaf  fhe  definifion  of  fhe  benchmark  suife  musf  be  a  joinf  efforf  of  researchers  and  pracfifioners  in 
fhe  field  of  publish/ subscribe  sysfems.  Therefore,  we  infend  our  proposal  as  a  stimulus  for  discussion,  and 
possibly  a  sfarfing  poinf  for  fhe  formulafion  of  a  comprehensive  and  widely  accepfed  benchmark. 


Acknowledgments 

The  work  of  fhe  aufhors  was  supporfed  in  parf  by  fhe  Defense  Advanced  Research  Projecfs  Agency,  Air 
Force  Research  Laboratory,  Space  and  Naval  Warfare  Sysfem  Cenfer,  and  Army  Research  Office  under 
agreemenf  numbers  F30602-01-1-0503,  F30602-00-2-0608,  N66001-00-1-8945,  and  DAAD19-01-1-0484.  The 
U.S.  Governmenf  is  aufhorized  fo  reproduce  and  disfribufe  reprinfs  for  Govemmenfal  purposes  nofwifh- 
sfanding  any  copyrighf  annofafion  fhereon.  The  views  and  conclusions  confained  herein  are  fhose  of  fhe 
aufhors  and  should  nof  be  inferprefed  as  necessarily  representing  fhe  official  policies  or  endorsemenfs, 
eifher  expressed  or  implied,  of  fhe  Defense  Advanced  Research  Projecfs  Agency,  Air  Force  Research  Labo¬ 
ratory,  Space  and  Naval  Warfare  Sysfem  Cenfer,  Army  Research  Office,  or  fhe  U.S.  Governmenf. 


6 


References 


[1]  G.  Banavar,  T.  D.  Chandra,  B.  Mukherjee,  J.  Nagarajarao,  R.  E.  Strom,  and  D.  C.  Sturman.  An  efficient 
multicast  protocol  for  confenf-based  publish-subscribe  sysfems.  In  The  19^^  IEEE  International  Conference 
on  Distributed  Computing  Systems  (ICDCS  '99),  pages  262-272,  Austin,  TX,  May  1999. 

[2]  D.  J.  Barrett,  L.  A.  Clarke,  R  L.  Tarr,  and  A.  E.  Wise.  A  framework  for  evenf-based  software  infegrafion. 
ACM  Transactions  on  Software  Engineering  and  Methodology,  5(4):378-421,  Ocf.  1996. 

[3]  A.  Campailla,  S.  Chaki,  E.  Clarke,  S.  Jha,  and  H.  Veifh.  Efticienf  tittering  in  publish-subscribe  sysfems 
using  binary  decision  diagrams.  In  Proceedings  of  the  23th  International  Conference  on  Software  Engineering, 
pages  443-452,  Toronto,  Canada,  May  2001. 

[4]  A.  Carzaniga,  D.  S.  Rosenblum,  and  A.  L.  Wolf.  Design  and  evaluafion  of  a  wide-area  evenf  nofificafion 
service.  ACM  Transactions  on  Computer  Systems,  19(3):332-383,  Aug.  2001. 

[5]  G.  Cugola,  E.  Di  Niffo,  and  A.  Euggeffa.  The  JEDI  evenf-based  infrasfrucfure  and  ifs  application  fo  fhe 
developmenf  of  fhe  OPSS  WEMS.  IEEE  Transactions  on  Software  Engineering,  9(27):827-850,  Sepf.  2001. 

[6]  E.  Eabref,  H.  A.  Jacobsen,  E.  Llirbaf,  J.  Pereira,  K.  A.  Ross,  and  D.  Shasha.  Pilfering  algorifhms  and 
implemenfafion  for  very  fasf  publish/ subscribe  sysfems.  In  ACM  SIGMOD  2001,  pages  115-126,  Sanfa 
Barbara,  CA,  May  2001. 

[7]  L.  Opyrchal,  M.  Asfley,  J.  Auerbach,  G.  Banavar,  R.  Shorn,  and  D.  Sfruman.  Exploiting  IP  mulficasf 
in  confenf-based  publish-subscribe  sysfems.  In  J.  Svenfek  and  G.  Coulson,  edifors.  Middleware  2000, 
number  1795  in  LNCS,  pages  185-207,  New  York,  NY,  Apr.  2000.  Springer. 

[8]  D.  S.  Rosenblum  and  A.  L.  Wolf.  A  design  framework  for  Infernef-scale  evenf  observafion  and  nofifica¬ 
fion.  In  Proceedings  of  the  Sixth  European  Software  Engineering  Conference,  number  1301  in  Lecfure  Notes 
in  Computer  Science,  pages  344-360.  Springer- Verlag,  1997. 

[9]  E.  W.  Zegura,  K.  Calverf,  and  M.  J.  Donahoo.  A  quanfifafive  comparison  of  graph-based  models  for 
infemef  fopology.  lEEE/ACM  Transactions  on  Networking,  5(6),  Dec.  1997. 


7 


